The present disclosure relates generally to motion capture and in particular to motion capture using body-mounted cameras.
Motion capture refers generally to techniques for modeling movement of a body during a specific activity (e.g., running, jumping, dancing) starting from observation of an actual subject performing the activity to be modeled. These activities usually involve coordinated movements of the subject's limbs, head, and/or other body parts. In a traditional marker-based optical motion capture system, the subject wears a special suit that incorporates retroreflective or light-emitting markers at particular points on the subject's body (e.g., near the joints or at specific points on the limbs, torso, or head). Cameras set up at various locations around the subject record the movement of the subject as the subject performs the desired activity; for three-dimensional (3D) modeling, multiple cameras need to be observing the subject at any given time. Triangulation is used to recover the 3D position of these markers in space, and the 3D marker positions, in turn, are used to fit a skeletal model to the observed motion. Animated characters (which might or might not look like the original subject) can then be rendered from the skeleton.
While such systems can produce highly accurate results, they are generally expensive and also limiting as to where motion can be captured. Reliable identification of the markers requires controlled lighting and generally is only possible indoors. Further, since the cameras generally do not move, a large number of cameras may be required if the motion takes place over a large area.
For less intrusive motion capture, “marker-less” methods have been proposed. Marker-less methods most often use regular video cameras with simple (e.g., chromakey) backgrounds to reconstruct a voxel representation of the body over time and then fit a skeletal model to the voxel representations. Recent studies [Corazza, S. et al., Analysis Biomedical Engineering 34, 6, 1019-1029 (2006); Corazza, S. et al., IEEE Transactions on Biomedical Engineering 57, 4, 806-812 (2010)] suggest that with a sufficient number of cameras and favorable imaging conditions, the accuracy of marker-less methods can rival that of traditional optical motion capture. As with optical motion capture, however, these systems generally require costly setups with many synchronized video cameras.
Both marker-based and marker-less systems can be classified as “outside-in,” in that they both rely on sensors mounted in the environment and passive, if any, markers on the body. By definition, this requirement restricts their use to laboratory environments or closed stage settings because the capture space has to be instrumented with the sensors. This also restricts the size of the space where the capture can take place.
Another class of motion-capture systems uses an “inside-out” approach, relying on sensors on the body to recover the 3D pose. This allows for increased portability, including use in outdoor spaces.
One example of an inside-out system, developed by Raskar and colleagues [Raskar, R. et al., ACM SIGGRAPH (2007)], uses photo sensors worn by the subject as active markers. Raskar's system relies on measuring the spatio-temporal light modulations produced by multiple LED transmitters placed in the environment that emit Gray-coded patterns. The subject wears receiver modules equipped with RGB photo sensors, which are tasked with decoding (demultiplexing) the observed patterns and thereby directly determining the 3D spatial location of the subject. This system, however, still requires transmitters in the environment, making it only marginally more portable than more traditional optical motion capture setups.
Another type of inside-out system relies on an electro-mechanical exoskeleton suit worn by the subject. The suit includes embedded lightweight rods that articulate with the subject's bones. Potentiometers at the joints measure the angular rotation of the rods, and the measured rotations are converted to joint angles using calibration data. Such systems, while directly capable of measuring the motion of the subject, are intrusive and uncomfortable to wear, particularly for extended motion capture sessions.
Still other examples of inside-out techniques rely on other types of sensors worn by the subject. Various sensors have been used, including ultrasound, inertial measuring units (IMUs), and tri-axial accelerometers. Inertial motion capture systems measure the rotation of body parts in the world using accelerometers and gyroscopes. These systems are portable and can be taken outside; however, they are only able to measure the orientations of body parts, not the motion of the body in the world. In addition, they often suffer from substantial drift over time, making them practical only for capturing relatively short motions.